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ABSTRACT 

This paper focuses on the design of an assessment plan that 
can accurately measure the impact of using World Wide Web-based deliveries to 
increase performance. Key trends in technology and training are reviewed, and 
effective assessment of online training deliveries is examined. The Virtual 
Business Training Center (VBTC) is an integrated business resource center that 
provides students with access to online market research, project management, 
and other project-based data analysis opportunities. It also functions as a 
business lab and virtual training for business partners. The authors used this 
unique virtual project environment to facilitate the access of project data by 
students (retrieval, analysis, evaluation), re-post as work in progress onto a 
Web-based environment for business partners’ participation, then edit or move 
onto the next project phase. Current research results indicate that the Web- 
based environment enhances access to both volume and timeliness of data and 
information. This virtual training platform also speeds up analysis and 
contributes to more effective project completion. The contribution to project 
success was measured by adherence to schedule, agreed-upon deliverables, and 
resource allocation. (Contains 14 references.) (MES) 
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Abstract: The Virtual Business Training Center (VBTC) is an integrated business resource center 
that provides students with access to online market research, project management, and other 
project -based data analysis opportunities. It also functions as a business lab and virtual training for 
business partners. We use this unique virtual project environment to facilitate the access of project 
data by students (retrieval, analysis, then evaluation), re-post as Work in Progress onto web-based 
environment for business partner's participation, then edit or move on to the next project phase. 

Our current research results indicate that the web-based environment enhances access to both 
volume and timeliness of data and information. This virtual training platform also speeds up 
analysis, and contributes to more effective project completion. The contribution to project success 
was measured by adherence to schedule, to agreed-upon deliverables, and to resource allocation. 



Introduction 

A key online business relationship exists when an organization, facilitator, and learner interact within a 
technology platform to develop workplace performance. These technology tools can help learners (both students 
and employees) target and reach their highest performance objectives quickly and with reduced trainer support. In 
project based classes there are exciting opportunities for problem solving to be facilitated by advanced and 
interactive technology interventions. New knowledge can be gained by exploring unique applications of technology 
platforms that facilitate project deliveries. Preliminary results indicate an increase in interaction and access 
(participation) of data usage, analysis of data, and facilitation work by nearly three-fold compared to traditional 
face-to-face project environments. Our study also reflects the number of interactions between team members and 
the facilitator increasing on average from 22 to 48 contacts in the web-based environment. Quality gains have been 
observed including an increase of at least 25% of required deliverables completed. Total project completion to 
schedule increased from 0% on time completion in traditional project delivery to 80% using the web-based 
environment. A summary of the baseline and web-based intervention environments are included in Research Design 
section under Background subheading of this paper. A developmental assessment effort can be used to study 
existing online web-based intervention platforms and their effectiveness in facilitating a project-based delivery. 

The focus of this paper is on the design of an assessment plan that can accurately measure the impact of using web- 
based deliveries to increase performance. The full discovery of this relationship requires a quick review of key 
trends in technology and training and then a full examination of effective assessment of online training deliveries. 

Growth Trends in Technology Based Training and Instruction 

There has been a tremendous explosion in the types and use of technology to deliver workplace and 
learning education objectives. Each technology platform has seen growth with the web-based training platforms 
growing fastest. Some important growth trends are highlighted by the following American Society for Training and 
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Development (ASTD) 1999 statistics; 30% of all training delivery types are technology deliveries, only 3% are high 
end interactive (EPSS) technology deliveries, the use of multimedia has risen from 12% to 22% the last two years 
(up to 81% with companies with progressive training strategies — designated as benchmark companies), British 
Open University and University of Phoenix alone has grown from 10,000 online students in 1990 to over 50,000 in 
1998 (ASTD 1999), and projected growth of technology based delivery for training for 2002 is 55% an increase of 
25% from 1999 (Salopek 1999). 

Assessment Trends in Training and Education 

There is a glaring disparity when you observe the rise of inclusion of technology in training/education 
while there exists a relatively low measurement of its effectiveness. The following statistics highlight some recent 
trends; 94% of surveyed companies assess training at Level 1 assessment which reflects no increase from 1996, 53% 
of surveyed companies assess training at Level 2 assessment, 29% increase from 1996, 32% of surveyed companies 
assess training at Level 3 assessment, 19% increase from 1996, 28% of surveyed companies assess at Level 4 
assessment, 25% increase form 1996, output data collected (number of people receiving training, number of training 
offerings, $ spent on training, etc.) consistently lags behind input data collected (productivity, customer satisfaction, 
return on investment, sales, waste reduction, etc.) by 20% in almost every category, 94% of survey companies use 
questionnaires to assess, a low of 40% use performance records (ASTD 1997). 

Assessment Challenges 

The technology is used as a set of intervention tools to develop significant performance improvements. 
International Data Corporation’s ^search Manager Ellen Julian expresses the challenges as a maturation of 
Information Technology (IT) infrastructures that can now accommodate web based training, high interest among 
companies to pilot online learning systems, and an increasing pool of knowledge to implement those technologies. 
As an example she cites the recent acceptance and use of state of the art online chat to facilitate online training and 
now it is already jumped to implementing live, real time video conferencing and collaboration over the Internet 
(Julian 1998). The following items outline the key challenges that educators and trainers face when integrating 
online technologies into their delivery systems: technology developments occur faster than adoption, technology 
adoption occurs faster than assessment, and the most immediate evaluation challenges in business and education 
involve focus on expertise and commitment 

New Knowledge Requirements 

New Knowledge is where many training facilitators and university faculty must pause and consider their 
actions. Are the online interventions being used effective? Do the online interventions improve performance areas 
targeted? Are the online interventions more effective for the majority of the learners? Can learning using online 
interventions be measured? Can application of learning be measured? Other questions can and should be addressed 
as innovative online technology tools continue to be introduced and requirements for employee performance 
development increases. The burden at this point is obviously shifting from what can be done using online 
technology platforms to which ones work and where are they best applied. Unfortunately this is a forgotten task of 
trainers and faculty, we plan and do and then plan and do again without gaining key knowledge of the effectiveness 
of our techniques or tools. Assessment represents the set of activities that moves from doing (implementing a new 
online component such as video conferencing) to studying the doing action and determining whether the 
intervention actually made an improvement in learning or application. If we have quantitative and qualitative based 
knowledge in a particular online intervention that did or did not improve learning or application then we have new 
knowledge - assessment’s ultimate goal. The process for effective gain of this new knowledge includes four steps. 

Planning an online intervention - This step is not an arbitrary action guided by what technology is available 
or most innovative but rather a structured and focused plan. After initial defining activities selection of a subsystem 
to improve takes place, the improvement effort exposes identifiable skills or knowledge gaps, and then a baseline 
needs to be taken. What level of skills do learners possess right now? What is the level of performance exhibited 
currently? What is the level of satisfaction that customers of this subsystem express? What is the current training 
intervention used to improve these areas (classroom deliveries, CBT, etc)? This balanced measure baseline will 
provide data that can be used later to determine whether a new and innovative online technology training 
intervention has actually made an improvement in skill and knowledge application and whether that imp rovement 
has had an impact on exhibited performance, customer satisfaction, and competitive position. The gap between the 




3 



existing performance and desired performance is the next piece of the plan to be examined. These planning 
activities include identification of all the interferences between the existing performance and the desired 
performance. Finding the most significant interference prepares us for the next stage of a new knowledge quest. 

Implementing an online intervention - Immediately after planning the training focus area that requires 
performance improvement selection of an appropriate online intervention is considered. The elimination of the 
significant interference becomes the criteria for selection of which online intervention will be best to try. At this 
point a match of needs, expertise and availability of technology can be maximized. A small intervention change 
should always be attempted first before major investments are taken and because we do not yet have knowledge of 
the effectiveness of the technology intervention. 

Assessing the technology intervention - Quantitative and qualitative examination of the intervention as it 
compares to the baseline. The design, collection, analysis, and evaluation of the data results will determine whether 
the online technology training intervention had the desired impact, did it work? 

Standardize the use of the online intervention - this step may also include the abandonment of technology 
interventions that do not make improvements in learning and application. Increasing retention of information that 
does not increase student/employee performance is not improvement and should not be used to determine 
technology intervention effectiveness. If an online technology intervention has been documented to improve a 
training output (learning and application) then the improved training intervention should be standardized. The new 
technology intervention should be implemented on a larger scale than the pilot. It can be used to deliver other 
knowledge or skills area needs (i.e. sales to software instruction) or used for all learners in a particular skill area. 



Framing the Assessment 

Assessment Problem Component 1 - Exhibiting Business Knowledge and Technology Skills 

The project based approach is a technique, pedagogy, and delivery of business info into university and 
training environments and an integral piece of improving the effective training of future business leaders. Project 
work in general provides an environment where learners exhibit a direct, hands-on application of business skills, 
knowledge and attitudes. Learners have the opportunity in this environment to develop key business skills that are 
difficult to deliver in the traditional classroom format. The project based approach functions as a “pull” system, 
much like successful and innovative production control systems that require parts to be delivered only as they are 
needed using a Just in Time (JIT) system. Pulling the business content requirement and technical skills into the 
project/classroom at exactly the time learners need those enablers optimizes the benefits of experiential learning 
project work. It also increases the range of applications of various business skills. The question of what type of 
enabler (skill intervention) is needed is matched with what best enables the successful completion of the project 
component. The project also is the significant criteria for what should be introduced (knowledge intervention) as 
preparation. The timing of the required training intervention is determined by the immediate needs of the project 
team in resolving project problems. A comprehensive project based program will require the understanding and 
usage of relevant knowledge and expertise. The real time nature of projects should eliminate the expended time lag 
between the receipt of theoretical information and its implementation in solving problems. 

Key training highlights of this Just in Time / Pull approach include; less Work In Progress - stockpiling 
information eliminated, quick response to problems - completion of projects require skill and knowledge gaps to be 
addressed immediately before progressing further, better quality - duplication of skill, knowledge, and attitudes is 
known and avoided by students and faculty (do it right the first time), better market response - actual business 
projects are as dynamic as business environment, student confidence - building cycles of success, supports practice, 
increases level of participation, develops levels of leadership and teamwork. 

The Pull system approach - delivery and pedagogy highlights for faculty and students include; 
opportunities to test and refine change and improvement theories, requires exhibited knowledge, raises awareness 
level of business success and potential, raises awareness level of team success, raises awareness level of professional 
and personal success, presents challenges requiring more than one set of skills or knowledge bases, requires depth 
and of content and application expertise, ream centered, problem solving emphasis, learn how to learn emphasis/self 
directed learning 

Assessment Problem Component 2- Making Projects Work 



A comprehensive project-based approach puts tremendous pressure on the use of project management skill 
of both learners and facilitators. In addition, the pull system success is directly hinged to successful project 
completion. Unsuccessful project completion compares to covering 7 of 22 chapters in a textbook based course, the 
textbook range is the basis for required body of knowledge. A completed project meets the body of knowledge 
requirement for project-based approaches. With this in mind, executing the project effectively and efficiently is 
paramount to learning and success. Our recent work with technology platforms specifically focuses on more 
effective project management. An innovative use of online technologies is not just an alternative to be used in 
project based business classes, it is a better platform. An online platform in a training setting will increase 
participation of learners, provide more opportunities for critical thinking, enhance communication between team 
members, enhance communication between business sponsor/faculty facilitator/student team, increase probability of 
achieving required deliverables, quicker responses to faculty interventions, project closer to planned timeline 
schedule, more communication between team members, clearer exhibition to business skills and knowledge, more 
cohesive group work, project completion closer to planned specifications, earlier interaction with facilitator/faculty, 
earlier interaction with business sponsor, and greater number of outside experts involved with project. 



Research Project Description 

This applied research project will focus on examining solutions to practical proble ms experienced in 
project based classroom deliveries. This Type I Developmental Research will emphasize the study of a specific 
online technology platform and its use and will produce a research output that articulates lessons learned from 
analyzing this online technology platform in use to facilitate business projects in the workplace and college 
environments. 

Research Questions 

Can online platform improve access (increased volume - quantitative) to resources (information, data, team 
members, etc.)? Can online platform improve interaction (increased effectiveness - qualitative) with and among 
resources (team members, facilitator, sponsors, data analysis, etc.)? Can improved access and interaction through 
online platform improve project management success? 

Are students (more?) satisfied with use of online technology to facilitate team project activities? Are 
students (more?) satisfied with the results of online technology toward project completion? Are students (more?) 
satisfied with online technology as a learning environment? Is faculty more satisfied with student participation and 
demonstrated expertise? 

Assessment/Research Hypothesis 

Can a high end and interactive online training environment enhance access and interaction of team member 
activities and improve project completion? 

Project Statement 

To improve the high end interactive online training environment as measured by frequency of access and 
interaction and successful project completion. 

Operational Definitions for Project Statement/Hypothesis 

• Access 

1. Information - schedule, work in progress, problems, questions 

2. Expertise - technical, analysis, project management 

3. Software applications - spreadsheets, scheduling, statistical analysis, etc. 

4. Planning - with sponsors, team, facilitator 

• Interaction (textual or interactive multimedia using technology tools from remote locations) 

1 . Synchronous - real time discussions, presentation of information or data, planning activities, 

consensus/decision making sessions, work sessions analyzing or stratifying data, and project status 
reporting. 
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2. Asynchronous - separate time activities that include online discussions, presentation of information or 
data, planning activities, consensus or decision making sessions, work sessions analyzing or stratifying 
data, and project status reporting. 

3. Just in Time interventions - the opportunity for interventions with timely expertise. Facilitators can 
arrange skill based or knowledge based interventions as they observe online team activities. 

• Successful project completion 

1 . Deliverables - meeting agreed upon outcomes at agreed upon quality levels that become the 
specifications of any project. 

2. Adherence to Project Schedule 

Research Design 

The proposed research project will model a Type 1 developmental study that aligns with more traditional 
evaluations where the primary research focus will be on the product/program evaluation and not the development 
process. The development process will be described and analyzed while the online platform, as a product, will be 
evaluated. This research project proposes to study the impact of an existing (recently developed) online platform 
(product/program) on student performance with actual business projects. The product of this research will clearly be 
a report on lessons learned from the use of a unique online platform to facilitate and impact student teams’ effective 
and efficient work on business projects. 

Background 

The evaluation of the web-based intervention will be established within a quality operations course. This 
Quality Principles course has 3 main learning components; review of foundation principles of quality theory, 
supplemental discussion issues in quality applications today (team building, organizational culture, leadership, etc.), 
and a business project simulation. The business simulation is an actual Continuous Improvement project completed 
for a Midwest organization that has been transitioned into an interactive project simu lation. The project is very 
structure in the problem solving process as well as its required outputs, schedule, and level of accuracy. Students 
work in teams on the project by moving through the Plan-Do-Study-Act cycle. The cycle is broken down into a 7- 
phase problem solving approach with required procedures and techniques for data gathering, analysis, consensus, 
and decision-making. The project schedule identifies specific outputs to be completed through out the problem 
solving sequence including flow charts, control charts, Cause and Effect diagrams, and Pareto Diagrams. The 
prompt and interaction cycle is flexible but typically follows a facilitator prompt, intra team interaction, project 
schedule management, facilitator intervention, Work in Process outputs reviewed, team or individual interaction 
with the facilitator, project outputs, next prompt from facilitator. 

The baseline class completes the entire project in the classroom (computer lab) with minimal outside of 
class interaction. A traditional business meeting format includes the team interaction as well as the 
facilitator/faculty participation. The meetings include work activities such as general feedback, tool development 
instruction, process prompts, and software application instruction. 

The web-based intervention is integrated in an identical section of the quality operation course. The three 
main learning components remain the same with the exception being the web-based delivery and project interaction 
of the Continuous Improvement project. The project is facilitated entirely online including all feedback, tool 
development instruction, software instruction, etc. Lotus Domino and Course Info’s Blackboard platforms provide 
threaded discussion boards, real time chat function, e-mail, file posting capability, and online work space usage. 

Data to be gathered 

Primary data to be gathered on frequency of access and interaction and successful project completion 
includes: attribute data counts on number of uses of online features (chat function, desk top video conferencing, 
discussion board postings, etc.), variables data on amount of time using online features as % of time on project with 
team and as individual contributor., attribute data on number of contacts with project data, team members, analysis 
features, communication, and project status information., attribute (yes/no) data on meeting project specified 
deliverable milestones, and attribute (yes/no) data on meeting project specified schedule milestones. 

Sub study data to be gathered includes: qualitative data on student satisfaction with online platform, 
qualitative data on student satisfaction with project completion, qualitative data on student satisfaction with online 
learning environment, qualitative data on faculty satisfaction with student’s demonstrated expertise and participation 
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Methods of Data Collection 



Descriptive research method - This research method will be used to describe what is the impact on using 
online interventions for access, interaction, and project completion. We will summarize data (basic central 
tendencies - i.e. mean number of chat room intervention usage, variations - i.e. variation of measured time 
interacting with data posted on web site, percentages - i.e. percent of students satisfied with use of web based 
environment to drive business projects, and correlation between variables - i.e. measure the relationship between 
those students who indicate a positive attitude toward web based platform usage and number of times interacting 
with video conferencing feature) and review it for explanations specific to successful online interventions for 
effective project management. The descriptive research will also provide validation on the level of effectiveness 
interactive web based platforms can have on project completion. 

Method of Data Analysis and Instruments 

Descriptive statistic analysis - analysis will include graphics that aid in data stratification, consensus 
building, analysis, and decision-making. Instruments will include surveys and tally sheet/observation. This analysis 
will be used to represent the respondent profile, basic levels of satisfaction with online platform, baseline levels for 
project activity usage of online technology components (chat feature, desktop video conferencing, etc.), descriptions 
of use of each technology component (chat feature for brainstorming, desk top video conferencing for team data 
analysis, etc.), and measurement of intervention impact on team project performance. Instruments will include 
spreadsheet software (Excel), statistical software (SPSS), and presentation software (PowerPoint). 

Cross tabulation Analysis- analysis will include graphics that aid in the interpretation of key relationships 
in attitudes, technology usage, and technology effectiveness. Instrument will be a survey. Software will include 
statistical software (SPSS) and presentation software (PowerPoint). 

Variation Analysis - analysis will include attribute and variables data recording, control chart design, and 
control chart interpretation. The instrument will be observation/tally sheet, participant responses on checklist, and 
survey. This analysis will be used to represent the performance of data collected over time including counts of each 
tech component usage during project activity, time spent with each tech component during the project activity, 
determine special cause variation existence, etc.. The performance tracking will be used as baseline and as 
measurement to determine whether online interventions improved access, interaction, and project completion. 
Software will include Statistical Process Control Software (SQCPak). 

Attitude Pattern Profile - analysis will include qualitative interpretation of discussions including attitude 
toward online technologies, use of technologies in education, use of technology in project management, team 
building, etc. The instruments will be interviews and focus group. This analysis will be used to verify and expand 
information collected and analyzed from surveys. 
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